Learning longer-term dependencies via grouped distributor unit
نویسندگان
چکیده
منابع مشابه
Learning Longer-term Dependencies in RNNs with Auxiliary Losses
We present a simple method to improve learning long-term dependencies in recurrent neural networks (RNNs) by introducing unsupervised auxiliary losses. These auxiliary losses force RNNs to either remember distant past or predict future, enabling truncated backpropagation through time (BPTT) to work on very long sequences. We experimented on sequences up to 16 000 tokens long and report faster t...
متن کاملLearning Long Term Dependencies via Fourier Recurrent Units
It is a known fact that training recurrent neural networks for tasks that have long term dependencies is challenging. One of the main reasons is the vanishing or exploding gradient problem, which prevents gradient information from propagating to early layers. In this paper we propose a simple recurrent architecture, the Fourier Recurrent Unit (FRU), that stabilizes the gradients that arise in i...
متن کاملMorphology and Longer Distance Dependencies
This paper illustrates the advantages of laboratory research into SLA by describing two studies of acquisition of second language syntax. The first addresses the question of whether human morphological abilities can be understood in terms of associative processes or whether it is necessary to postulate rule-based symbol processing systems underlying these skills. We demonstrate that acquisition...
متن کاملHigh-speed routers design using data stream distributor unit
As the line rates standards are changing frequently to provide higher bit rates, the routers design has become very challenging due to the need for new wire-speed router’s network processor (NP) unit. Typically, designing new NPs could take a long time and is very costly. In this work, we are presenting a new approach in high-speed routers design. Our approach is to use a data stream distributo...
متن کاملLearning Hidden Markov Models to Fit Long-Term Dependencies
We propose in this report a novel approach to the induction of the structure of Hidden Markov Models (HMMs). The notion of partially observable Markov models (POMMs) is introduced. POMMs form a particular case of HMMs where any state emits a single letter with probability one, but several states can emit the same letter. It is shown that any HMM can be represented by an equivalent POMM. The pro...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
ژورنال
عنوان ژورنال: Neurocomputing
سال: 2020
ISSN: 0925-2312
DOI: 10.1016/j.neucom.2020.06.105